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ABSTRACT 

A complex system is considered in its latter stages of development. 
N mission trials have been observed, each resulting in a success or a 
failure. Each failure occurs in one of k failure modes. For each 
failure mode that is observed action is taken to attempt to correct 
that type of failure. The probabilities of correcting the various 
failure modes are known. After corrective action is completed attempts 
to estimate the current reliability, without further sampling, are 
made. A brief historical summary of this problem to date is given. 

Justification for assuming a prior distribution on the failure 
modes is discussed and the posterior distribution of the parameters 
is developed. An intuitive measure of the current reliability is 


stated and certain properties of this random variable are developed. 
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1. INTRODUCTION 

In the latter stages of development and testing of a complex 
system it is reasonable to assume that a certain number of mission 
trials have been effected, and of these some number resulted in the 
failure of the mission. An intuitive measure of the reliability of 
the system at this point would be the ratio of failures to the total 
number of tests conducted. After analysis of the failure data however, 
it is conceivable that the reasons for each failure of the mission 
could be detected and some corrective action taken against this type 
of failure. After taking corrective action the natural step would be 
to continue testing and make statements about the reliability of the 
revised system on the basis of new data. 

If the cost of additional testing is high, if additional test units 
are not available, or if time is a prohibitive factor, further testing 
may not be feasible. At this point Corcoran, Weingarten and Zehna oe 
pose the following problen: "Assuming that we have confidence in our 
knowledge of the effectiveness of the contemplated corrective action 
which may be taken on observed failure modes, how should we use the 
results of the first N tests to draw inferences about the current reli- 
ability?" The problem is specifically structured in this manner. Let 
N tests be conducted. Each of the tests results either in a successful 
performance of the mission or ina failure. Each failure results from 
a failure of one or k possible modes. The probability of a successful 
mission is q? the probability of a mission failing by the jth failure 
mode is q. where i ranges from one to k. The q's themselves are 


unknown; however, since each trial must result in a success of some 


type of failure it may be noted that 


q, = il. 


tT -1* 


ifF=o 


Similarly if we let N, be the number of events of the ith type observed 


where No is the number of successful missions then 


k 

+ 

' Ns = N 
ih 1 
iczo 


It is also assumed that there is a known conditional probability as of 
correcting the ith ¢silure mode given that it occurs. 

Based on the above formulation Corcoran, Weingarten, and Zehna [1] 
define the following random variable as a measure of current reliability: 


Oifw = 0 
oF where y, = i 
i 


a if N > QO 
i i 


This is stated to be an intuitive measure of the reliability since it 
adds a weighted amount of the failure probability of each observed 
failure mode to the initial reliability. The expected value of p* may 
be computed and is referred to as the "mean reliability.'' This quantity 


is shown to be: 


N 
4 [1 - (1-q,) } 


k 
ELp*] = q. +) : 


a 
i=l 
Since p* is a random variable it is customarily not estimated. However, 
the variance of p*¥ tends to zero as N tends to infinity; therefore any 


estimate of EL p*] can be said to asymptotically estimate p*. The above 


[1) 


authors ~ postulate seven estimators of ELp*] and discuss their relative 





merits from a standpoint of bias. The last two estimators, quoted 
here for convenience, are conservative (underestimate ELp*]) and are 
asymptotically unbiased. The last estimator, Pz» is the more con- 


servative of the two but is also consistent for EL p*]. 


> 1 
Nu EP ig) =i se 
Oumme\- i 
Pas ae /. sie where z = 
S N ~ 0 otherwise 
i=l 
k N,-l 
+Samiy ot hon _|a ifN >0 
Es — te 2 " where y i i 
|= 


O if N, =0 
i 


The bias for estimator p. is shown not to exceed .Ol1 for an N of 25 


6 
and a ae as large as one third. 

Larson [3] considers the conditional distribution of the reliability 
(conditioned on the outcome of the test} and demonstrates its probability 
mass function. Unfortunately while the mass function is known, the 
actual values the random variables take on are unknown since they are 


functions of the q, ‘8. A functional lower bound on the true reliability 


is shown to be: 


plz = £|A] = P Cg = fla) plz = glial 


where the conditioning event A is the event that N ae a” 0, f is 


1 

a function of the observable random variables and g is some function of 

the unknown parameters. Pile 2 £ {A} is a conditional multinomial 

probability statement and P[Z 2 gla] is a statement from the distribution 

of true reliability conditioned on sample results. The expression 

PLZ 2 gla] can be evaluated for a certain class of g functions}; thus a 
siegeson, H. J., Conditional Distribution of True Reliability after 


Corrective Action (US Naval Postgraduate School. Technical Report/Research 
Paper No. 61, 1966) p. 10. 
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lower bound on the reliability can be obtained if the appropriate 
conditional multinomial statement can be derived. 

This approach has been considered by the present investigator but 
abandoned due to the compounding complexity in the probability state- 
ments. The class of functions of the parameters (q,‘s) is restrictive 
since the P[Z 2 gla] must be capable of evaluation. No suitable 
functions of the observable random variables have been found that lead 
to direct evaluation of the multinomial statement. The conditional 
distribution of the sample can be obtained but it becomes exceedingly 
complex as the number of failure modes increases. As an example if we 


consider only three modes of failure (Nj oN and N,) and the conditioning 


Z 


> 0, N. = 0} then the conditional probability is 


event A = Yi 0, No 3 


as follows: 


PLN = a, N= db, N= N-(atb) [A] 


——: N-(atb) 
sada aga q2 


tat Mi t N N N 
at b! [N-(atb)]! U(q +4, +4.) +4. -(q_ +4,) s(autas) ] 


O<a< N-2 
for 
0O<b < N-a 
5 0 Otherwise 


It would seem that any direct attempt to derive the distribution of 
some function of the N's is hopeless. Even if the distribution of 
the multinomial statement can be derived for some class of f and g 
functionsthere is no guarantee that these choices of functions will 
produce a useful bound on the reliability. The right hand side of the 


confidence interval statement is multiplicative and hence both 


P({g2 £|A] and P[Z 2 g lA] must be capable of simultaneously yielding 
m 
high numerical values at the sample point. This of course places 


further restrictions on the class of functions available. 


\O 


2. BAYESIAN VIEWPOINT 

In both previously cited works (1, 3] the probabilities of failure 
for each of the various modes are considered unknown parameters. In 
certain cases it may be feasible to assume a prior distribution on the 
failure modes themselves. As a specific example if the system consists 
of k components in logical series one failure mode may be associated 
with each component. Since earlier developmental history should be 
available on each component it is reasonable to assume that something 
is known about the reliability of each. On the other hand if each mode 
of failure is to be some type of failure (electrical, mechanical, etc.), 
it still may be reasonable to assume that there exists some prior 
knowledge of these failure rates. 

In considering priors in general there are several requirements 
that must be met. The marginal range of each of the random variables 
must be the interval zero to one. Each of the component reliabilities 
will be close to one; hence the distribution chosen for a failure mode 
must be capable of lumping a good percentage of its probability near 
zero. The prior should lead to easy computation if possible or at 
least be tractable. 

Three forms of prior distributions were considered. 

G Os q= bs l/kK  i=-**k 
(ly) £ (qyaasie? aei= ‘ 


Q Q eeey Q 
1? 2? k 
< 0 Otherwise 


m 
Gq, 4, tt 059,5 bS 1/k i=l,++*k 
(2) £ (q,34 sd) = f 7 
Q7 2892" °° Qk lL 2 


0 otherwise 
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q q 
Oo)? k 
B52 Qa 2 Sy k 


In all three cases q represents the probability of a successful mission 
° 
and q, represents the probability of the jth failure mode as before. 
ae 


The G.‘'s are appropriate constants. 
a 


The first two priors assume independence between the various 
failure modes. Unfortunately the posterior for each of these distri- 
butions is difficult to demonstrate. The k fold integral appearing 
in the denominator of each posterior is non-integrable and numerical 
techniques would have to be resorted to if it were to be evaluated. 
This was not deemed worthwhile since the third form of the prior is 
actually a more general representation than the first two. It is true 
that (3) disallows independence between failure modes but no strong 
argument can be made for this independence. The last prior will be 


discussed more thoroughly and developed in the next section. 


11 


3. PRIOR DISTRIBUTION OF COMPONENT FAILURES 

If we consider a series of N tests as before with qd, 3s the 
probability of a success on any given trial and a iw] 2, °°k, “as Che 
probability of a failure by the ith failure mode, then the probability 
of observing N, successes and N. failures, i=1,2,°°*k, in the respective 


failure modes will follow the multinomial probability law. 


Thus, 
Ne N Ny Ny 
eee eee — ‘@) _ 
PLN GN ote °N faosdyo° ay) 5 et te o> ee | 
L=o 
k 
N. = 0,1,2,°°° \ = 
A »41,2,°°°N Ly N 
i=o 


Raiffa and Schlaifer! suggest using a prior of the same functional 
form as the distribution of the sample. This procedure leads to prior 
(3) depicted in the last section. The advantages of this choice for a 


prior will become apparent in the succeeding development. 


Th 
us, - ai m)-1 m~1 
PL q q,°°°q J = G q 4 q » cag, 
0771 k fe) 1 k 
k 
\ = 1 2 0 
where a qd. q. 


1=0O 


shall be the chosen prior. Making use of the usual fact that a density 


integrated over its range must equal one allows us to evaluate G. 


Rateean H. and Schlaifer, R., Applied Statistical Decision Theory 
(Boston: Graduate School of Business Administration Harvard University, 
1961) p. 47-49. 
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k es m,~L 
i = | (1 ze y a = 5 dq eeedg 
Gl J ad 44 i a. 
t= 
i 
m -1 ra] 7) 
l ce ‘ 
J ---f 1 | qd eee k-1 
q, Woy k-1 
i.) a k 
. = : v m -1} aD 
i. —_" oO 
; (ieee?) a pe 
q i=l 
=o 
gq 
N betti = : 
ow letting x= ca 
ie 2 
D4 
i=l 
an, = mtm-1 m)-1 Oy i 
. Poe (il - > 4) qd) eee a1 dq eee “wy 
te 
Mh m -i1 lel 
i x, (1 - x) dx, 
q 
k=o 


It may be noted that the last integral in the above expression is just 


a Beta integral with parameters Mm, and m integrated over its range. 


Hence: 


l -l 

: ae , "0 
J x (1 - x) = ENE: m) 
Oo 
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By a continued iteration of the above procedure it may be shown that; 


uf 
is Btn. , m) Bia, he mt) B(m, : mn +m ) 


22 k k-1l 


—- 


coe BC,» m +o,+m + coe a 


Noting that 


'(m,) I’ (tg) 


Bim , m) = 
ee I'(m +m ) 
k oO k 
cn 
(ma a a toeee m,) o P| 
0 
T(m ) C(m,) ame (nm, ) Tl oT (mm) 
. 1=o ‘ 


Hence the complete prior is given by: 


em 
7 age ae, m@ ., =o-l m -l 
f ( eee ) = —1=0_______ o qd - ose G 
Q ,Q ,74* Q Go? q\? qd. k . l k 
gue k n Tv @) 
_— 1 
1= 
k 
where \ 


ma | andiques0 1-S07 haa i 
i 


i 


fe 
i 
.@) 


Several properties of this prior will be needed at a later point; for 


convenience they will be derived or substantiated at this time. 


(a) Marginal Distribution of the q 's 
1 
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k 


Consider 
\ ( 2 ) 
\o" = ° ° m a m Ww 1 a“ l 
f —————— oer a q coe Gg dap... dq 
1 mT (m ) q, qk 
i 
1=0 
Iterating this integral as in the previous case; 
k 
ie aS 
ii . | 
Se y 
pee te ore. Le see oo. 
or oe Ete Semen 82? tov inl a hay 
1 n PF @,) 
1=o 
* mj- a. oe eee mateo ea 1 
q q, 
k 
Letting _ * om 
i=o 
m,-l m-m -l 
lr (m) 1 
f£ (q) = q Glee a4 
— : iF (m,) r (m - ees - 2 
m -1 m-m-l 
1 - l ) 
= qd, (1 - q, O< a 1 
B(m,; m - m,) 
-~ 0 otherwise 


> By symmetry arguments the fay marginal is of the same for, 


therefore; 
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m -l m-m -j 
1 i 
*0 1)! ie ae oe (1 -aap 
i B(m > O- m.) 
i i 
* £ .(q, m.; 9 See) 0 2 ae= 
B i a 


Thus the marginals of the prior are Beta distributed with the appropriate 


parameters as shown. 


For this reason and for convenience in notation 


the prior itself will be referred to as a multivariate Beta. 


(b) Expected Value of the Q 's with Respect to the Prior Marginals 








i 
Sl 
E = qf )d 
OP)" Ma : (q) dq 
fe) i 
’ l 
ry m-m - 
de ar 
mo m A 
Hence 
Bim + 1, m -m ) 
1 1. me 
E(Q.) = * 
1 B(m,, m - m) m 
(c) Variance of the Q 's 
1 
l 
Se 1 
2 1 r m.tl m m. 
E(Q “) = | oe ace =" q) 
i B(m, , m- m_) ©] 
i 
m. (m; + 1) 
m(1l + m) 
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dq 


Thus 


var(q.) = £(Q.%) - B°(Q) 
1 | k : 


1 
mm - =? 
m°(1 + m) 


(d) Joint Density of Q., Q 
i 


m-l mol m-m,-m,-1 
f ( ) —___—-@ q : q J (1-q_-q ) 
q.>» WIJ = -q.-q. 
QQ. 1 jj Cl (m.) T (mm) T (a-m-my 2+ J ij 
1 Jj 1 j i j 


J 
q + q <1 q 2 0 q 2 0 
i j . i 
(e) Covariance of Q , Q. 
i 
L leq 
e r 1 ; m 
l(m) | 3 ~~? : 
£(Q,Q ) Se a i 
J (m.) [T (m) T (m-n.-n. 
>D? ¢ > (m-m ;? Gin Fan 
mom j-m_~1 
(lL - @ -q) J dq dq 
1 J L j 
Let q 
x a 
i 
q, 


Then integrating as before 
, ¥ 
mej 


} = 
oe m (1 + m) 
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And 


- E(Q,) E(Q.) 
Cov (2.9) - B22.) , j 
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4. POSTERIOR DISTRIBUTION OF THE PARAMETERS 
In the process described previously the distribution of the sample 


was given by 


, n Ny N 
PLN N *oeN g coed ] = ree Et a qd - gq oa qd ke 
<a k'"o ‘*k ad o L k 


Utilizing the prior described in section three and applying Bayes rule 





we get; 
2 4 -1 
: Notm-+ = Ny+m,-1 Ad 
- 2 ke 
P qd ee °q N 9°? oN = 
k Oo k 1 ca - 
te) . f me Nh -1 k Notm-1 

|} eee axe le ; , 
J . a, WP pal.d dq, ++ -dq, 


qq q i=1 


Noting the similarity between this integral and that previously seen 


in section three gives 


N tm -l N, +m. -1 
[ lr (N + m) = kk 
P coe eee = qd eee 
%, a. IN NJ r (N +n yooeD (N, +m. ) re} 4. 
O Oo k k 
k 
4 ~ey b fey 
Where N = a N. and m a m, as before 
L=o i=o0 


Except for suitable changes in parameters this distribution can be 
recognized as the multivariate Beta of the previous section. This 
facility in handling the posterior is, of course, one of the reasons 
for choosing this particular distribution for a prior. Of course a 


prior must meet more requirements than just facility in use. The 


Je 


multivariate Beta in question does satisfy the intuitive demands set 
forth earlier. A logical method for selecting the parameters of the 
prior (ms) will be given in Appendix I. The assumption of this prior 
is equivalent to a form of "linear squashing" described by Gooa* in his 
work on the estimation of probabilities. 

As has been noted above the posterior distribution is itself a 
Beta, hence the properties of the posterior may be stated by analogy 
from the previous results. For future convenience these properties 


will be listed below. 


(a) Posterior Marginal 


N +m,” 1 N+m- (N +m.) -1 
¢ (alu en, ¢§ (1-q) 
Q ae k B(N.tm , Ntm - (N,+m,)) 
i i> @ oe 
(b) Marginal Mean 
N. + m 
E(Q |N yeeN) = 
io N +m 
(c) Marginal Variance 
Z 
(N+m) (N +m) - (N.+ m_) 
Var( IN N.) = iil 1 1 
Q, 3” k 


(N +m)? (1 +N +m) 


(d) Conditional Covariance of Q, Q. 
1 J 


(N, +m.) (N, +m) 
Cow(QkOn |S. iN, = - ———— 
i). Se (N +m)? (1 4+N +m) 


Good, I. J., The Estimation of Probabilities, an Essay on Modern 
Bayesian Methods (Cambridge: Research Monograph No. 30, The M.I.T. Press, 
1965) p. 24. 
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5. RELIABILITY 

Corcoran, Weingarten and Zehna [1] define the current measure of 
reliability, p*, that was stated in section one of this paper. The 
estimation problem is considered to have taken place prior to the 
observation of the N tests. Averaging over all possible outcomes of 
the N tests to produce the "mean reliability" is justified on this basis. 
Larson [3] regards this of interest in the early phases of development 
before testing can take place but points out that the final reliability 
is actually a function of the outcomes of the N tests. This is true 
because whether or not a corrective attempt is made depends on observing 
the given failure mode. For this reason he developes the conditional 
distribution of true reliability (conditioned on the outcomes of the 
N tests) and shows that p*¥ is in fact the mean of this conditional 
distribution. This poses the question of how to properly interpret E(p*). 
Since p* is the mean of the conditional reliability it would seem 
unreasonable to average over all possible outcomes of the N tests (some 
of which are known not to have occurred) to obtain E(p*). Larson 
considers it more reasonable to attempt to estimate p* rather than E(p*). 

There would seem to be other avenues of approach open. Consider 
the original question of attempting to make probability statements about 
a complex system after some period of testing and applying corrective 
action. The discreteness of the conditional reliability function could 
be interpreted as an anomaly arising from simplification in the logical 
statement of the problem. The uncertainty in the actual reliability 
arises from two sources: the uncertainty in the failure modes themselves 


and the uncertainty of correcting a type of failure if it occurs. In 


21 


some cases it may be feasible to treat the uncertainty in the failure 
modes by assuming a prior distribution on the A, 8: Partial justi- 
fication for this procedure was given in section two. 

It does appear attractive to treat the reliability as conditioned 
on the outcomes of the N tests. At least at that point in time we know 
which modes of failure it is possible to remove. With the above 
considerations in mind the following scheme is proposed. Let N tests 
be conducted, the results of which are h. successes and N, failures in 


i 


the respective failure modes, i = 1,2,°°° k. The probability of a 


success is qd? the probability of the failure of the ig) node is De, 
. 

where 

k k 

\ 1 \ 

= oe 

ae, Let 

i=o i=o 


The maximum number of failure modes that can occur is k; it is not 
necessary that each mode be observed however, and a sample will in 
general demonstrate less than k failure types. If we let BL, be the 
probability of correcting the ith ¢ailure mode given that it is observed 
and b. be the unconditional probability of correcting each failure mode 


it follows that 


The following is stated as an intuitive measure of the reliability 

after corrective action. This is of course the same measure defined 
by Corcoran, Weingarten and Zehna [1]. The interpretation given to 
it here, however, differs from that of the above paper. The actual 


sampling is considered to have taken place and R is conditioned on this 
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outcome through the b 's. 


L 


k 
Req +h 
74 Lo Pde 


i=l 


k k 
ya,+ ). 
= > — b. 
gb id; 
i=l i=l 
k 
= . b ae 1 
L+)} ( : ) q, 
i=1 


If we are willing to assume a distribution on the q, § then R is 
a function of random variables and is hence itself a random variable. 
The remainder of this study will be devoted to the development of the 
properties of this measure of reliability with respect to the posterior 
distribution developed in section four. R is t.:e weighted sum of 
dependent Beta distributed random variables. It is not possible to 
obtain the convolution of independent Beta's, hence there is little or 
no hope of deriving the distribution of R. 

Now 7 

E(R) = 1 +) (b, - iets 8 


i=l 


Noting from section four 


. em. 
E(q ) = ee 
N+m 
—k N. a m; 
E(R} = 1 +) (b, -1) 
N +m 


i=l 


23 


Using the variance and covariance formulation developed in section four, 
let 


(N “- m) (Nn, +m) 


oO SS = 
ii 
: (N+? @+n4 YD 
2 
(N +m) (No + m.)-(N,4m ) 
ee ee | 8 
ii (N -” (N + m + 1) 


Now let V equal the matrix 
| i w1,2, -+<:k 
nee. a 
1j 41 ‘2 log eos E& 


4 
and a equal the vector 


be ss ee orcas 5 -) 
Then the variance of R is the quadratic form 


Var(R) = a’ Va 

Thus although the distribution of R cannot be shown its mean and 
variance are known. These parameters are functions of the N's and 
hence are uniquely determined for each sample point. In an informal 
sense the mean and variance of a random variable can be said to "estimate" 
the random variable and hence E(R) provides some indication of the 
corrected value of the reliability. In the above case it may be noted 
that the variance of R tends to zero as N tends to infinity; hence the 
density of R "lumps" at the mean as N increases. In this sense E(R) 
may be said to asymptotically estimate R as N increases. This fact may 
be of little use however since the N's of interest can be expected to 


be small. 
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It is feasible that the higher moments of the distribution could 
be obtained. While these moments may be complex they should be functions 
of the N's and hence capable of being evaluated. anesities all of the 
moments would of course be equivalent to obtaining the actual distri- 
bution. 

One last avenue of approach may be worthy of mention. It was 
initially hoped that the formulation in this study would lead to the 
ability to make confidence interval statements about the reliability. 
Except for the loose bound obtainable through Chebychev's inequality 
this has not been realized. The question of the existence of some 
limit theorem has not been thoroughly investigated. This does not 


appear promising but it may warrant some further consideration. 
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APPENDIX I 


SELECTION OF PRIOR PARAMETERS 


One possible method of selecting the parameters of a prior 
distribution would be to fit a set of prior means and variances to 
the distribution. With the multivariate Beta in question however 
there are insufficient parameters (k + 1) to do this. The following 
method suggested by silwer* is given. 

We use the last k parameters to fit the means, then the remaining 


parameter, mo is used to obtain a least squares fit to the prior 


variances. 
Let k 
(1) me) m 
i 
i=o 
Then 
= 
eee y= -—— i Saioh ae 
i m. 
m.(m - m.) 
(2) AG 3.2 = i = 1,2,-+-k 
: m (1 + m) 


Substituting equation (1} into equation (2) 


E(q.) [1 - B (q,) ] 
L L 


Var = 
(a, m+ 1 


Let var(q,) be the prior estimated variance of ne 





Silver, E. A., Markovian Decisions Processes with Uncertain 
Transition Probabilities or Rewards (M.I.T., Interim Technical 
Report No. 1, 1963) Appendix C, pp. 177-179. 
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Hence we want to choose m to minimize 


k 
Ss =) (var(q.) - var(q.))° 
i=l 
k 2 
=_— E(q,) [1 - E(q,)] 
Ss =) Var(q ) - eg 
j=] : m+ il 


Setting d(SS)/dm equal to zero we get the following for a least squares 


value of m equal to mn. 


k 


as 2 
as [eq] [1 - E(q )J 
im. i - 

k 


y var(q_.) E(q ) L1 - E(q )} 
if i 


i=1 


- 1 


> 
fl 


and hence 


m =m E(q_) i eel,2;° 2 ee 
i i 
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